篇首语:本文由编程笔记#小编为大家整理,主要介绍了Flink on Yarn三部曲之三:提交Flink任务相关的知识,希望对你有一定的参考价值。
现在Flink、Yarn、HDFS都就绪了,接下来实践提交Flink任务到Yarn执行;
接下来提交的Flink任务是经典的WordCount,先在HDFS中准备一份文本文件,后面提交的Flink任务都会读取这个文件,统计里面每个单词的数字,准备文本的步骤如下:
SSH登录CDH服务器;
切换到hdfs账号: su - hdfs
下载实战用的txt文件:
wget https://github.com/zq2599/blog_demos/blob/master/files/GoneWiththeWind.txt
创建hdfs文件夹: hdfs dfs -mkdir /input
将文本文件上传到/input目录: hdfs dfs -put ./GoneWiththeWind.txt /input
准备工作完成,可以提交任务试试了。
./bin/yarn-session.sh -n 2 -jm 1024 -tm 1024
bin/flink run ./examples/batch/WordCount.jar \\-input hdfs://192.168.50.134:8020/input/GoneWiththeWind.txt \\-output hdfs://192.168.50.134:8020/wordcount-result.txt
bin/flink run -m yarn-cluster \\-yn 2 \\-yjm 1024 \\-ytm 1024 \\./examples/batch/WordCount.jar \\-input hdfs://192.168.50.134:8020/input/GoneWiththeWind.txt \\-output hdfs://192.168.50.134:8020/wordcount-result-1.txt
bin/flink run -m yarn-cluster \\-yn 2 \\-yjm 1024 \\-ytm 1024 \\./examples/batch/WordCount.jar \\-input hdfs://192.168.50.134:8020/input/GoneWiththeWind.txt \\-output hdfs://192.168.50.134:8020/wordcount-result-2.txt